Technical Field
The present disclosure relates to the storage and retrieval of temporally organized computer network traffic, such as packet data.
Background Information
A computer network is a communications network allowing attached devices, such as nodes, to exchange network traffic. The traffic exchanged among the nodes on the network is broken up into packets. Software and hardware capture devices may capture these packets and use various formats for storing the packets. Common to all formats are standard attributes describing the data encapsulated in the packets and the network nodes where the traffic is visible. Because of its simplicity, flexibility, and portability, the packet capture (PCAP) format has become the de facto standard for packet storage.
Modern computer networks support hundreds of nodes transferring multiple gigabytes of data each second, which can generate a large amount of network traffic in a very short amount of time. Many capture devices record the network traffic in chronological order as packets are extracted from the network. Modern disk access times and processing power helps speed the search for data, but physical limitations remain a limiting factor. To avoid discarding data, writing newly captured packets may be given priority access to resources, thereby reducing the resources available for search and retrieval tasks. One approach for searching and retrieving packet data is to perform a sequential (linear) search of the data, comparing each packet to a filter describing the target criteria, which may be time consuming and inefficient. Indexing may be used to reduce search time requirements at the cost of significant overhead to create and store indices. However, these indices themselves may constitute a relatively large amount of information that must be searched to locate packets of interest, thereby reducing the benefit of the indexing itself. Therefore, a technique to reduce the index search is needed.